Cognitive  Modeling  and  Task  Analysis: 
Basic  Processes  and  Individual  Differences 


F49620-98- 1-0031 

FINAL  REPORT 
(Period  12/15/97  -  11/30/99) 


Phillip  L.  Ackerman,  Principal  Investigator 


School  of  Psychology 
Georgia  Institute  of  Technology 
274  5th  Street 
Atlanta,  GA  30332-0170 


Submitted  to: 

Air  Force  Office  of  Scientific  Research 
AFOSR/NL 


801  North  Randolph  Street,  Room  732 
Arlington,  VA  22203-1977 


This  is  a  Final  Report  prepared  for  the  Air  Force  Office  of  Scientific  Research. 


PXIC  QUALITY  INSPECTESD^ 


20000417  214 


REPORT  DOCUMENTATION  PAGE 


AFRL-SR-BL-TR-OO- 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  respon 
sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection 
other  aspect  of  the  collection  of  information,  including  suggestions  for  reducing  this  burden  to  Was 
Reports,  1215  Jefferson  Davis  Highway.  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Manage 
Washington,  DC  20503. 


3iib 


existing  data 
imate  or  any 
I  Operations  and 
(0704-0188), 


1.  AGENCY  USE  ONLY  (Leave  blank) 

2.  REPORT  DATE 

14  Feb  00 

3.  REPORT  TYPE  AND  DATE  COVERED 

Final  15  Dec  97  to  30  Nov  99 

4.  TITLE  AND  SUBTITLE 

5.  FUNDING  NUMBERS 

Cognitive  Modeling  and  Task  Analysis: 

Basic  Processes  and  Individual  Differences 

F49620-98-1-0031 

6.  AUTHOR(S) 

Phillip  L.  Ackerman 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

8.  PERFORMING  ORGANIZATION  REPORT  NUMBER 

Georgia  Tech  Research 
Corporation 

400  Tenth  Street  N.W., 
Atlanta,  GA  30332-0420 

Ackerman- 02 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSORING/MONITORING  AGENCY  REPORT  NUMBER 

Air  Force  Office  of 
Scientific  Research 
AFOSR/NL 

801  North  Randolph 
Street,  Room  732 
Arlington,  VA  22203- 
1977 

F49620-98-1-0036 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 
Unlimited 


12b.  DISTRIBUTION  CODE 


13.  ABSTRACT  {Maximum  200  Words) 

The  goal  of  the  current  research  program  was  to  select  a  critical  subset  of  task  conditions  and  individual 
differences  variables  and  evaluate  the  joint  effects  of  these  variables  on  task  performance  in  a  complex- skill 
environment.  The  subset  of  task  conditions  selected  were  those  that  involve  basic  processes  of  working  memory,  task 
monitoring,  and  differential  loads  on  spatial  reasoning  and  speed  of  perceiving.  The  subset  of  ability  determin^ts 
of  performeuice  included  broad  content  abilities  (spatial  and  math  abilities) ,  perceptual  speed  abilities  (including 
complex  perceptual  speed,  scanning,  memory  and  pattern  recognition) .  A  complex  task  platform  was  adopted  that 
simulates  many  aspects  of  the  job  of  an  air  traffic  controller.  Memipulations  of  task  component  demands  and 
communication  channel  reliability  were  used  to  evaluate  both  overall  effects  and  moderating  effects  on  ability  - 
performance  relations.  Manipulations  of  task  characteristics  proved  to  be  highly  effective  in  changing  the  mean 
levels  of  task  perf ormeince .  However,  critical  ability  measures  were  demonstrated  to  be  robust  predictors  of 
performance,  even  under  substantially  altered  task  constraints.  Implications  of  these  findings  are  discussed  with 
respect  to  extant  theory  and  empirical  research.  In  addition,  brief  discussion  is  provided  regarding  the  potential 
applications  of  those  findings  for  operational  environments. 


14.  SUBJECT  TERMS 
Individual  Differences 
Abilities 
Working  Memory 


15.  NUMBER  OF  PAGES 
38 


16.  PRICE  CODE 


17.  SECURITY  CLASSIFICATION 
OF  REPORT 
Unclassified 


18.  SECURITY  CLASSIFICATION 

19.  SECURITY  CLASSIFICATION 

OF  THIS  PAGE 

OF  ABSTRACT 

Unclassified 

Unclassified 

(Rev.2.89) 


20.  LIMITATION  OF  ABSTRACT 


orm  298 
Prescribed  by  ANSI  Std.  Z39-18 
298-102 


Summary 


The  goal  of  the  current  research  program  was  to  select  a  critical  subset  of  task 
conditions  and  individual  differences  variables  and  evaluate  the  joint  effects  of  these  variables 
on  task  performance  in  a  complex-skill  environment.  The  subset  of  task  conditions  selected 
were  those  that  involve  basic  processes  of  working  memory,  task  monitoring,  and  differential 
loads  on  spatial  reasoning  and  speed  of  perceiving  and  decision  making.  The  subset  of  ability 
determinants  of  performance  included  broad  content  abilities  (spatial  and  math  abilities), 
perceptual  speed  abilities  (including  complex  perceptual  speed,  scanning,  memory  and  pattern 
recognition).  Measures  of  working  memory  ability  were  administered  to  evaluate  whether 
additional  convergence  could  be  found  between  task  component  performance  measures  and 
individual  differences  predictors  of  performance.  A  complex  task  platform  was  adopted, 
which  simulates  many  aspects  of  the  job  of  an  air  traffic  controller.  Initial  training  was 
provided  to  familiarize  trainees  with  the  task,  and  to  provide  a  baseline  assessment  of  ability 
determinants  of  performance  in  each  of  the  task  components.  Subsequent  manipulations  of 
task  component  demands  and  communication  channel  reliability  were  used  to  evaluate  both 
overall  effects  (e.g.,  mean  performance  level  differences)  and  moderating  effects  on  ability  - 
performance  relations.  The  analyses  of  data  from  this  study  address  the  manipulation  effects 
on  both  overall  performance  and  on  correlations  between  predictors  and  performance  criteria. 
In  general,  manipulations  of  task  characteristics  proved  to  be  highly  effective  in  changing  the 
mean  levels  of  task  performance.  However,  critical  ability  measures  were  demonstrated  to  be 
robust  predictors  of  performance,  even  under  substantially  altered  task  constraints. 

Implications  of  these  findings  are  discussed  with  respect  to  extant  theory  and  empirical 
research.  In  addition,  brief  discussion  is  provided  regarding  the  potential  applications 
of  these  findings  for  operational  environments. 
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I.  Introduction 

This  research  project  involves  the  continuation  of  two  lines  of  theoretical  and  empirical 
research  that  have  been  of  historical  and  current  concern  to  the  U.S.  Air  Force.  The  first  line 
of  research  has  been  the  determination  of  abilities  that  are  of  critical  importance  for  the 
prediction  of  individual  differences  in  skill  learning,  both  at  initial  phases  of  training,  and 
later  on-the-job  performance.  This  has  been  a  central  issue  for  selection,  training,  and 
classification  purposes,  even  before  formal  the  existence  of  a  separate  U.S.  Air  Force  (e.g., 
the  extensive  work  conducted  by  the  U.S.  Army  Air  Forces  Aviation  Psychology  Research 
Program,  under  the  direction  of  John  Flanagan  (see  Flanagan,  1948;  Guilford  &  Lacey,  1947; 
Melton,  1947).  Over  the  past  50  or  so  years,  efforts  have  been  devoted  to  developing  and 
validating  taxonomic  representations  of  cognitive  and  intellectual  abilities.  However,  more 
research  is  still  needed,  given  the  complexity  of  human  cognitive  processes,  and  the  limited 
resources  that  have  been  available  for  their  investigation. 

The  second  line  of  research,  task  and  job  demands,  originally  started  by  the  same 
research  group  (e.g.,  Fitts),  and  is  generally  regarded  as  the  beginning  of  Engineering 
Psychology.  Specifically,  the  attention  of  psychologists  during  and  after  World  War  II 
represented  a  shift  from  “fitting  the  person  to  the  job”  to  “fitting  the  job  to  the  person” 
(McCormick  &  Sanders,  1982).  The  original  impetus  for  this  work  was  the  high  rate  of 
aviation  accidents  that  had  been  “determined”  to  be  due  to  pilot  error.  Work  by  Paul  Fitts 
and  his  colleagues  determined  that,  in  many  cases,  the  design  of  cockpit  displays  and  controls 
(especially  the  lack  of  standardization),  was  inherently  incompatible  with  the  limitations  and 
capabilities  of  pilots  —  especially  under  stressful  combat  conditions.  Three-pointer  altimeters 
could  not  be  read  accurately  or  rapidly,  or  identical  control  knobs  could  not  be  differentiated, 
except  by  taking  visual  attention  away  from  the  windscreen.  Thus  the  information  needed  by 
the  pilots  for  safe  and  effective  operation  was  frequently  unavailable  or  was  available  only  in 
a  degraded  fashion.  The  approach  to  solving  such  problems  has  been  one  of  analyzing  both 
task  requirements  and  the  basic  limitations  and  capabilities  of  the  human  operator. 

These  two  lines  of  research  come  together  when  it  is  realized  that  the  successful 
operation  of  a  larger  system  (e.g.,  pilots,  weapons  officers,  support  staff,  AW  ACS  controllers, 
and  so  on),  is  determined  by  a  constantly  shifting  interplay  between  the  individual  differences 
in  abilities  among  the  personnel,  the  amount  of  training  provided,  and  the  specific  demands  of 
the  tasks  that  the  personnel  are  called-upon  to  perform.  Changes  in  the  pool  of  applicants, 
such  as  a  shift  to  higher  or  lower  mean  levels  of  abilities,  impacts  the  importance  of  the 
selection  and  training  systems.  When  the  applicant  population  includes  many  more  highly- 
talented  individuals,  selection  systems  become  more  important.  Conversely,  when  the 
application  population  fails  to  deliver  highly-talented  individuals,  more  expense  and  effort  is 
needed  in  the  training  system.  There  are,  however,  natural  limits  to  these  tradeoffs  —  no 
amount  of  training,  given  extant  methods  and  current  success  rates  will  develop  an  unqualified 
trainee  into  a  fully  skilled  pilot  or  AW  ACS  controller.  In  addition,  task  design  interacts  with 
both  the  selection  and  training  functions.  To  the  degree  that  intellectually  demanding  task 
components  can  be  off-loaded  or  automated,  the  demands  lesson  for  both  selection  and 
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training.  However,  history  tells  us  that  new  technological  developments  have  generally  had 
the  opposite  path.  That  is,  new  technologies  typically  increase  the  cognitive  load  on  the 
operator,  rather  than  decrease  it. 

These  considerations  lead  to  the  current  project  -  which  focuses  on  basic  research 
issues  on  the  dimensions  of  task  characteristics  that  determine  cognitive  load,  and  on  the 
individual  differences  in  ability  dimensions  that  determine  the  likelihood  of  successful  training 
and  skilled  performance.  The  research  described  in  this  report  is  just  a  small  part  of  a  larger 
effort  by  many  researchers  in  the  field  of  psychology.  It  draws  on  contributions  from 
cognitive/experimental,  engineering,  educational,  and  differential  psychology.  A  full  historical 
treatment  of  this  extensive  literature  is  beyond  the  scope  of  this  report,  but  it  cannot  be 
stressed  too  much  that  the  work  builds  on  a  foundation  that  has  been  established  by  many 
others  -  including  several  key  researchers  who  previously  worked  for  the  U.S.  Air  Force  over 
the  past  50  years  (e.g.,  R.  L.  Thorndike,  P.  Fitts,  L.  G.  Humphreys,  J.  P.  Guilford,  J.  Adams, 
E.  A.  FleisWan,  R.  Christal,  P.  Kyllonen,  and  many  others). 

II.  Current  Research 

The  goal  of  the  current  research  program  was  to  select  a  critical  subset  of  task 
conditions  and  individual  differences  variables  and  evaluate  the  joint  effects  of  these  variables 
on  task  performance  in  a  complex-skill  environment.  The  subset  of  task  conditions  selected 
were  those  that  involve  basic  processes  of  working  memory,  task  monitoring,  and  differential 
loads  on  spatial  reasoning  and  speed  of  perceiving  and  decision-making.  The  subset  of  ability 
determinants  of  performance  included  broad  content  abilities  (spatial  and  math  abilities), 
perceptual  speed  abilities  (including  complex  perceptual  speed,  scanning,  memory  and  pattern 
recognition).  Measures  of  working  memory  ability  were  administered  to  evaluate  whether 
additional  convergence  could  be  found  between  task  component  performance  measures  and 
individual  differences  predictors  of  performance.  A  complex  task  platform  was  adopted, 
which  simulates  many  aspects  of  the  job  of  an  air  traffic  controller.  Initial  training  was 
provided  to  familiarize  trainees  with  the  task,  and  to  provide  a  baseline  assessment  of  ability 
determinants  of  performance  in  each  of  the  task  components.  Subsequent  manipulations  of 
task  component  demands  and  communication  channel  reliability  were  used  to  evaluate  both 
overall  effects  (e.g.,  mean  performance  level  differences)  and  moderating  effects  on  ability  - 
performance  relations.  The  analyses  of  data  from  this  study  address  the  manipulation  effects 
on  both  overall  performance  and  on  correlations  between  predictors  and  performance  criteria. 
Implications  of  this  research  are  discussed  with  respect  to  extant  theory  and  empirical 
research.  In  addition,  brief  discussion  is  provided  regarding  the  potential  applications 
of  these  findings  for  operational  environments. 

Critical  Abilities  for  Performance 


In  the  current  research  project,  and  based  on  our  previous  AFOSR-sponsored  research 
(Kanfer  &  Ackerman,  1990;  Ackerman  &  Kanfer,  1996;  Ackerman,  1999b),  we  have 
developed  theory  and  conducted  extensive  empirical  research  with  an  aim  toward  developing  a 
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useful  classification  of  abilities  that  are  critical  for  the  prediction  of  individual  differences  in 
performance  during  and  after  skill  acquisition.  In  concert  with  other  U.S.  Air  Force  research 
(e.g.,  Kyllonen,  1985,  1994;  Woltz,  1988)  and  the  extant  literature  on  human  abilities  (e.g., 
see  Carroll,  1993  for  an  extensive  review),  we  have  identified  several  key  ability  classes  that 
have  been  demonstrated  to  have  substantial  validity  for  predicting  individual  differences  in 
performance  on  skill-learning  tasks.  Each  of  these  abilities  is  briefly  discussed  below. 

General/Content  Abilities.  General  intellectual  abilities  have  perhaps  the  most 
substantial  basis  for  predicting  individual  differences  in  learning  tasks  -  starting  with  the 
early  research  of  Binet  &  Simon  (1905)  regarding  the  prediction  of  academic  performance. 
Initial  research  and  theory  (e.g.,  Spearman,  1904)  suggested  that  general  intelligence  was 
amorphous  and  indivisible.  However,  by  the  1920s  and  1930s,  it  had  been  clearly  established 
that  general  intelligence  was  not  all-encompassing  in  terms  of  either  basic  underlying 
individual  differences  (e.g.,  see  Kelly,  1928;  Thomson,  1939;  Thurstone,  1938),  or  in  terms  of 
building  optimal  models  for  predicting  performance  in  specific  domains.  So-called  “group 
factors”  or  “content  abilities”  (e.g.,  spatial,  math,  verbal)  were  reliably  found  when  large 
batteries  of  ability  tests  were  subjected  to  factor  analysis.  Moreover,  examination  of  these 
content  factors  revealed  that  validity  in  predicting  individual  differences  in  performance 
depended  in  part  on  a  match  between  the  content  of  the  predictor  ability  and  the  content  of 
the  criterion  task.  That  is,  when  trainees  were  required  to  learn  highly-spatially  demanding 
tasks  or  jobs,  tests  of  spatial  ability  were  generally  more  highly  correlated  with  criterion  task 
performance  than  were,  say,  verbal  ability  measures  (e.g.,  see  Guilford  &  Lacey,  1947). 

It  is  possible  to  identify  dozens  of  group  factors,  but  as  shown  by  Snow,  Kyllonen, 
and  Marshalek  (1984),  dividing  the  ability  sphere  into  Spatial,  Math,  and  Verbal  abilities  is  a 
valid  simplification  for  categorization  purposes.  In  addition,  this  breakdown  of  the  structure 
of  abilities  provides  a  simplifying  scheme  that  can  be  adapted  to  particular  broad-based 
prediction  paradigms  (when  the  researcher  wishes  to  generalize  beyond  a  narrow  range  of 
criterion  tasks).  We  refer  to  these  categories  as  content  factors,  because  at  least  on  a  surface- 
level  analysis  of  task  requirements,  the  content  of  the  tests  describes  the  stimuli  upon  which 
the  individual  must  work  with  to  solve  the  test  problems  (i.e.,  words  for  Verbal  content, 
figures  for  Spatial  content,  and  numbers  for  Math  content) 

A  final  general  ability  has  also  been  subjected  to  a  great  amount  of  discussion  in  the 
past  10  years  or  so.  In  one  fi'amework  Working  Memory  (a  construct  fi'om  the  experimental 
psychology  literature)  can  be  thought  of  as  attentional  capacity  (e.g.,  see  Baddeley,  1986). 
Recent  discussion  has  also  portrayed  working  memory  as  essentially  the  same  thing  as 
reasoning  ability  —  which  is  central  to  the  conceptualization  of  general  intelligence  (over  and 
above  respective  content  abilities),  see  Kyllonen  &  Christal  (1989,  1990).  Working  Memory 
has  been  implicated  as  critical  for  predicting  individual  differences  in  task  performance, 
especially  on  complex  tasks.  As  such,  a  consideration  of  Working  Memory,  as  a  supplement 
to  the  content  factors  will  be  incorporated  into  the  empirical  study  described  below. 
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Percentual  Sneed  Abilities.  Previous  research  in  our  laboratory  has  demonstrated  that 
perceptual  speed  abilities  may  provide  a  critically  important  supplement  to  measures  of 
general  and  broad  content  abilities,  in  the  prediction  of  individual  differences  in  performance 
during  skill  acquisition  (e.g.,  Ackerman,  1988;  1990;  1992;  Ackerman  &  Kanfer,  1993; 
Ackerman,  Kanfer  &  Goff,  1995).  According  to  Ackerman  (1988),  Perceptual  Speed  (PS)  is 
defined  as:  “In  the  language  of  skill  acquisition,  individual  differences  found  on  [perceptual 
speed]  tests  are  directly  attributable  to  the  speed  with  which  these  productions  can  be 
implemented  and  compiled  (e.g.,  see  Werdelin  &  Stjemberg,  1969).  (p.  190)”  It  turns  out 
that  a  more  extensive  analysis  of  extant  measures  of  PS  ability  suggested  that  the  tests  could 
be  taxonomized  on  a  number  of  (overlapping)  dimensions,  based  on  an  information-processing 
analysis  of  test  content.  A  previous  analysis  (Ackerman  &  Rolfhus,  1996)  suggested  the 
following  candidate  dimensions  of  underlying  test  demands:  (a)  Content  (spatial,  verbal, 
numerical),  (b)  Consistency  (consistent  information  processing  or  varied  information 
processing);  (c)  Novelty  (familiar  vs.  novel  stimuli);  (d)  Precision  (of  encoding  and  of 
responding);  (e)  Modality  (of  encoding  and  of  responding);  (f)  Memory  demands  (low  vs. 
high),  (g)  Scanning  vs.  single  items. 

While  it  would  be  theoretically  possible  to  create  a  completely-crossed  sampling  of 
tests  for  these  seven  dimensions,  the  exponential  number  of  tests  that  would  be  generated 
make  for  an  impractical  empirical  evaluation  (especially  inasmuch  as  tests  could  also  combine 
dimensions  -  such  as  a  test  that  had  mixed  spatial  and  verbal  content).  Instead,  in  a  study 
now  submitted  for  publication  (Ackerman  &  Cianciolo,  under  review)  we  attempted  to  sample 
broadly  from  most  of  these  dimensions,  in  the  hope  that  patterns  of  consistent  individual 
differences  would  emerge,  inductively,  across  the  various  dimensions.  In  all,  we  adapted  or 
created  15  tests  of  PS  abilities,  to  administer  along  with  6  extant  measures  of  PS  abilities,  12 
tests  of  content  abilities  (verbal,  spatial,  numerical),  and  a  set  of  Choice  and  Simple  RT 
measures. 

By  subjecting  the  PS  tests  to  a  factor  analysis,  we  established  that  there  is  a  general  PS 
factor,  but  there  were  also  three  lower-order  PS  factors  that  were  clearly  differentiable  in 
terms  of  the  information  processing  demands  of  the  tests.  Specifically,  one  factor  was 
dominated  by  tests  that  involved  recognition  of  simple  patterns  (e.g..  Canceling  Symbols  and 
Finding  e  and  ¥)  —  which  we  designated  as  PS-Pattern  Recognition  (abbreviated  PS- 
Pattem).  A  second  factor  involved  scanning,  comparison,  and  look-up  processes  (e.g.,  Name- 
Comparison  Test  and  Number  Comparison  Test),  which  we  designated  as  PS-Scanning.  The 
third  factor  was  best  identified  as  making  substantial  demands  on  working  memory  (e.g., 
Digit/Symbol,  and  Coding  tests).  We  identified  this  factor  as  PS-Memory.  A  complete 
representation  of  PS  ability  also  includes  an  additional  PS  factor  identified  in  previous 
research  (called  PS-Complex,  because  the  tests  involve  both  traditional  PS  and  additional 
cognitive  components,  such  as  spatial  ability  and  estimation/interpolation,  and  heightened 
working-memory  loads;  see  Ackerman,  Kanfer,  &  Goff,  1995).  This  factor  structure  was 
replicated  in  a  later  study  (also  reported  in  Ackerman  &  Cianciolo,  under  review). 
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Based  on  our  research,  these  four  PS  factors  (PS-Complex,  PS-Memory,  PS-Pattem, 
and  PS-Scanning)  have  differential  associations  with  task  performance  across  skill  acquisition 
trials.  Some  (e.g.,  PS-Complex  and  PS-Memory)  are  especially  important  in  accounting  for 
individual  differences  in  performance  on  complex  skill  tasks,  while  others  (PS-Pattem  and  PS- 
Scan)  are  more  important  in  accounting  for  individual  differences  in  performance  on  simple 
skills. 


Psvchomotor  Abilities.  Although  not  considered  for  the  empirical  study  to  be 
described  below,  our  research  has  shown  that  there  psychomotor  ability  measures  can  also 
provide  incremental  predictive  validity  for  individual  differences  in  skilled  performance  - 
especially  for  tasks  that  are  routine  and  proceduralized  (see,  e.g.,  Ackerman  &  Cianciolo, 
1999;  under  review).  Ability  factors  for  Tapping,  Mirror  and  Maze  Tracing,  and  for  Serial 
Reaction  Time  have  been  reasonably  well  established  as  useful  predictors  after  extensive  task 
practice.  Most  of  these  ability  measures  are  not  regarded  as  useful  for  predicting  performance 
for  tasks  of  high  complexity,  and  there  was  no  a  priori  reason  to  expect  that  these  abilities 
would  be  sensitive  to  the  kinds  of  information  processing  load  manipulations  under  study  in 
this  project.  However,  for  future  research  on  less  complex  tasks  —  consideration  of 
psychomotor  abilities  is  certainly  warranted. 

Development  of  the  Experimental  Protocol 

Training  Phase.  The  main  goal  for  the  training  phase  was  to  provide  trainees  with  a 
reasonable  amount  of  experience  and  practice  on  a  complex  skill-learning  task  —  one  that 
makes  substantial  demands  on  the  critical  abilities  outlined  above  (namely,  content  abilities, 
working  memory,  and  at  least  some  of  the  PS  abilities).  The  training  phase  should  also 
provide  for,  after  practice,  relatively  stable  correlations  between  the  ability  predictors  and 
measures  of  task  performance.  The  underlying  criterion  task  also  was  required  to  have 
several  performance  components,  that  could  be  distinguished,  both  in  terms  of  mean  levels  of 
performance  (e.g.,  based  on  the  respective  information  processing  requirements  of  the  task 
components),  and  in  terms  of  ability-performance  components  (i.e.,  differential  correlations 
between  constellations  of  ability  predictors  and  component  task  performance).  The  Terminal 
Radar  Approach  Controller  task  (TRACON®)  provided  an  excellent  test-bed  for  our  purposes. 
The  task  is  sufficiently  complex  that  skilled  performance  develops  only  after  an  initial 
instruction  period,  followed  by  8-15  hours  of  task  practice.  Moreover,  there  are  three 
differentiable  task  components  embedded  in  the  task  --  namely  handling  overflights, 
departures,  and  arrivals. 

“Overflights”  are  planes  that  enter  and  exit  the  operator’s  airspace  at  cruising  altitudes. 
Trainees  are  required  to  acknowledge  these  airplanes  as  they  approached  a  boundary  Very 
high  frequency  omnidirectional  range  station  (VOR)  fix,  monitor  progress  through  the  sector, 
and  handoff  to  a  Center  controller.  “Departures”  are  planes  that  originate  at  one  of  the  four 
airports,  climb  to  a  cruising  altitude  and  are  to  be  handed  off  to  a  Center  controller. 

Operators  are  required  to  release  departures  from  airports,  evaluate  and  remediate  potential 
conflicts  as  the  planes  climb  to  a  cruising  altitude  and  turn  to  intercept  their  intended  flight 
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paths,  and  then  handoff  planes  to  the  appropriate  Center  controller.  “Arrivals”  enter  the 
operator’s  airspace  from  one  of  the  boimdary  VOR  fixes,  and  have  to  be  landed  at  a 
designated  airport.  Operators  are  required  to  direct  arrivals  onto  an  appropriate  heading  and 
altitude  to  provide  an  acceptable  handoff  to  the  appropriate  Tower  controller,  then  these 
planes  would  land. 

Extensive  data  on  the  TRACON  task  and  the  embedded  task  components  have  been 
collected  and  reported  in  the  literature  (e.g.,  Ackerman,  1992;  Ackerman  &  Cianciolo,  under 
review;  Ackerman  &  Kanfer,  1993;  Ackerman,  Kanfer,  &  Goff,  1995).  Overflights  have  been 
foimd  to  be  the  easiest  task  component.  Correlations  between  PS  measures  and  task 
performance  are  higher  for  this  component  than  for  the  other  two  components  —  mainly 
because  the  major  demands  on  the  trainee  are  to  scan  the  display,  monitor  for  conflicts  and 
remember  to  hand-off  the  plane  before  it  exits  the  controller’s  sector.  Arrivals,  on  the  other 
hand,  have  the  highest  amount  of  cognitive  demands  on  the  trainee.  Concomitant  with  the 
differences  in  task-component  difficulty  (and  differences  in  mean  performance  levels), 
differences  in  ability-performance  correlations  are  found.  Highest  correlations  with  general, 
spatial,  and  math  abilities  are  found  for  the  Arrival  component  of  the  task,  followed  by 
Departures,  and  then  Overflights.  These  differences  in  means  and  correlations  provide  an 
excellent  means  toward  triangulating  the  effects  of  changes  in  task  constraints. 

Testing  Phase.  In  order  to  assess  effects  of  task  demands  on  mean  performance  levels 
and  on  ability-performance  correlations,  one  can  change  the  constraints  of  the  overall  task, 
and/or  change  the  constraints  on  the  three  task  subcomponents.  The  approach  adopted  here 
was  to  parametrically  change  both  sets  of  conditions.  We  settled  on  two  manipulations  with 
these  goals  in  mind.  The  first  manipulation  was  an  overall  task  constraint  change.  By 
degrading  the  communication  link  between  the  controller  and  the  pilots,  we  believed  that  we 
would  increase  several  information  processing  demands  on  the  trainee.  The  standard 
condition  involves  error-free  communication  between  controller  and  pilot  —  every  command 
issued  to  a  pilot  is  read-back  without  error  and  compliance  is  assured,  unless  the  action  is 
beyond  the  capability  of  the  airplane.  A  degraded  (or  error-laden)  communication  channel 
introduces  a  stochastic  parameter  to  the  interactions  between  controller  and  pilot.  A  few 
examples  may  best  illustrate  this  condition.  In  some  cases,  a  command  issued  by  the 
controller  might  not  be  heard  by  the  pilot  -  and  thus  no  read-back  is  provided,  and  the  plane 
does  not  comply  with  the  instruction.  In  other  cases,  a  command  issued  to  one  pilot  may  be 
misunderstood  to  be  intended  for  another  pilot.  Under  this  scenario,  a  different  pilot  than 
intended  will  acknowledge  the  command,  and  comply  with  it.  (The  controller  might  instruct 
one  plane  to  descend  5000  feet.  Instead,  that  plane  will  continue  on  course,  and  another  plane 
will  descend  5000  feet.)  In  still  other  cases,  a  pilot  will  simply  reply  that  he/she  did  not  hear 
the  message,  and  will  request  that  the  controller  provide  it  again.  In  each  of  these  cases,  the 
controller  has  a  substantially  increased  cognitive  load  (mainly  in  terms  of  having  to  monitor 
the  communication  chaimel,  and  also  in  terms  of  increasing  the  frequency  of  visually  scanning 
the  radar  screen  to  make  sure  that  new  conflicts  do  not  arise  as  a  result  of  missed  instructions 
or  erroneous  reactions  to  other  commands). 
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The  second  manipulation  involved  changing  the  dominant  components  of  the 
TRACON  task  without  changing  the  overarching  structure  of  the  task.  That  is,  in  the  training 
simulations,  each  simulation  presented  the  trainee  with  28  planes  to  handle  —  16  arrivals  and 
12  overflights  and  departures.  By  creating  simulations  that  either  focused  on  Arrivals  or  on 
Overflights,  it  was  possible  to  change  the  respective  demands  on  content  abilities  (e.g.,  spatial, 
math)  and  on  perceptual  speed  abilities  (especially  PS-Complex  and  PS-Memory).  Under  the 
test  conditions,  an  “Arrival”  simulation  had  20  arrival  flights  and  no  overflights.  Under  the 
“Overflight”  test  conditions,  each  simulation  had  20  overflights  and  no  arrivals.  To  provide 
additional  measurements  of  performance,  and  to  assess  spill-over  effects  onto  other  task 
components,  we  filled-out  each  of  the  Arrival  and  Overflight  simulations  with  8  departures 
each.  The  test  conditions  were  similar  to  the  training  conditions,  in  that  both  had  28  total 
planes  to  handled  in  a  30-min.  simulation.  The  test  conditions  all  had  8  departure  flights, 
regardless  of  whether  arrival  or  overflights  were  present  in  the  simulations. 

Method 

Participants.  Ninety-four  adults  participated.  All  participants  were  native 
English  speakers,  had  normal  or  corrected-to-normal  hearing,  vision,  and  motor  coordination. 
The  sample  had  38  women  and  56  men,  =  21.00,  =  2.46,  range  18-30  years. 

Apparatus.  Pencil-and-paper  testing  (Session  1)  was  administered  in  a 
laboratory  with  prerecorded  instructions  and  directions  presented  over  a  public  address  system. 
Up  to  14  examinees  were  tested  at  a  time.  TRACON  training  and  testing  during  Sessions  2-7 
was  administered  on  Dell  and  IBM  Pentium  computers  with  17"  monitors  at  individual 
carrels. 

Ability  Tests.  Ability  tests  were  selected  on  the  basis  of  the  taxonomy  of 
critical  abilities  for  task  performance  described  above.  Tests  were  selected  to  assess  three 
broad  ability  factors  (Spatial,  Math,  and  Working  Memory),  and  four  narrow  ability  factors 
(PS-Complex,  PS-Memory,  PS-Pattem  Recognition,  and  PS-Scanning).  The  factors  and  the 
constituent  tests  are  presented  below. 

Spatial  Ability 

1.  Spatial  Analogy.  This  is  a  four-term  analogy  test  (A:B::C:D)  with  spatial  content. 

2.  Spatial  Orientation.  This  is  a  test  of  three-dimensional  visualization.  Examinees  are 
required  to  imagine  a  block  figure,  as  seen  from  a  different  perspective. 

3.  Paper  Folding.  In  this  test,  a  two-dimensional  projection  of  a  piece  of  paper  being 
folded  in  to  different  shapes  is  presented,  then  a  hole  is  shown  punched  in  the  folded 
paper.  The  examinee’s  task  is  to  visualize  where  the  holes  are  made,  when  the  paper 
is  shown  fully  unfolded. 
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4. 


Verbal  Test  of  Spatial  Ability.  This  is  a  test  of  image  generation  and  manipulation. 
Subjects  are  asked  to  close  their  eyes  and  imagine  the  items  described  verbally.  Then 
they  are  asked  a  multiple  choice  question  about  the  items  in  the  image. 

Example:  It  is  morning  and  you  are  facing  East  looking  at  the  sunrise.  You  walk 
forward  for  100  yards,  turn  left,  and  after  walking  another  50  yards  you  turn  about 
(i.e.,  turn  180  degrees).  In  what  direction  are  you  now  facing? 
a)  North  b)  South  c)  East  d)  West 

Math  Ability 

5.  Math  Knowledge.  This  is  a  wide  range  test  of  mathematical  knowledge,  from  simple 
computation  to  algebra,  geometry,  and  other  advanced  topics. 

Example:  What  is  the  product  of  16  and  10? 

a)  1.6  b)  .016  c)  160  d)  1600 

6.  Problem  Solving.  This  is  a  test  of  math  word  problems. 

Example:  You  are  traveling  at  the  rate  of  one  mile  per  minute.  What  is  your  speed  in 
miles  per  hour? 

a)  10  mph  b)  30  mph  c)  60  mph  d)  120  mph 

7.  Number  Series.  A  test  of  inductive  numerical  ability.  The  examinee  is  presented  with 
a  series  of  numbers,  and  asked  to  indicate  the  next  item  in  the  series. 

Example:  1  3  5  7  9  11  a)  11  b)  12  c)  13  d)  14 

Working  Memory  Tests 

8.  Computation  Span.  In  this  test,  examinees  hear  a  series  of  simple  equations  (e.g.,  “3  + 
2”).  They  are  instructed  to  write  the  answer  to  each  question.  The  are  also  instructed 
to  keep  a  running  memory  of  the  second  term  in  each  equation.  After  a  series  of 
equations,  the  examinees  write  down  the  list  of  second  terms  for  each  equation  in  the 
series. 

9.  ABCD  Order.  This  test  present  a  series  of  auditorially-presented  sentences  with  order 
information  regarding  4  letters  (A,  B,  C,  and  D),  and  order  information  regarding  sets 
of  letters.  The  examinee  must  use  the  order  information  to  place  the  letters  in  their 
appropriate  order.  Example:  “A  follows  B.”  “C  follows  D.”  “Set  1  follows  Set  2.” 
(The  correct  answer  is:  DCBA.) 

9.  Listening  Span.  In  this  test,  there  are  three  components  to  each  item.  The  first  part  is 
the  auditory  presentation  of  a  sentence.  The  second  part  is  a  visual  presentation  of  a 
question  about  the  sentence,  which  the  examinee  answers.  After  a  series  of  sentences 
and  questions  are  completed,  the  examinee  completes  the  third  part,  which  is  to  write 
down  the  last  word  of  each  sentence  in  the  series. 
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Perceptual  Speed  Tests 
PS-Complex  Tests 
1.  Dial  Reading  Test. 


A  B  CD  E 

Temperature  87.  92.  8.5  91.  81.5 


2.  Directional  Headings  Test.  In  this  test,  the  examinee  must  indicate  the  direction  indicated  by 
the  three  sources  of  information,  or  indicate  if  the  information  shows  a  conflict. 


(A) 

(B) 

(C) 

(D) 

(E) 

East 

South 

West 

North 

Conf 

(a) 

N 

t 

360 

s 

s 

* 

= 

(b) 

N 

* 

360 

= 

= 

= 

X 

(c) 

W 

270 

X 

* 

(d) 

E 

90 

= 

= 

= 

“ 

= 

(e) 

S 

t 

180 

= 

* 

* 

PS-Memorv  Tests 


1.  Coding.  In  this  test,  the  examinee  looks-up  the  a  letter  or  two-digit  codes  for  common  words. 


brown 

blue 

green 

brown 

grey 

red 

black 


COI/>R  CODE 

PRODUCT 

CODE 

black 

C 

Chair 

•< 

blue 

J 

lamp 

34 

brown 

L 

rug 

73 

green 

T 

stand 

52 

grey 

K 

stool 

4< 

red 

7 

table 

•  2 

yellow 

B 

vase 

27 

COlUMS  1 

M  T  X 

r 

C 

J 

8 

rug 

Column  2 

34  52  92  yS  46 

27 

86 

M 

T 

L 

F 

c 

Jt 

S 

lamp 

52 

92  73  46 

27 

86 

M 

Jt' 

L 

F 

c 

J 

S 

stand 

34 

92  73  46 

27 

86 

M 

T 

X 

F 

c 

J 

S 

rug 

34 

52 

92  46 

27 

86 

M 

T 

L 

F 

c 

J 

S 

stool 

34 

52 

92  73  46 

27 

86 

M 

T 

L 

F 

c 

J 

8 

table 

34 

52 

92  73  46 

27 

86 

M 

T 

L 

F 

c 

J 

S 

chair 

34 

52 

92  73  46 

27 

86 

2.  Digit/Svmbol  Test.  The  examinee  fills-in  the  numbers  corresponding  to  the  look-up  key. 


1 

d 

r 

H 

III 

n 

+ 

T 

4 

1^ 

_9 

Write  the  correct  number  below. 


[5 

= 

S] 

H 

r 

n 

n 

H 

X 

n 

= 

1 

1 

n 

=: 

r 

— 

d 

n 

a 

a 

5. 

0] 
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3.  Naming  Symbols  Test.  In  this  test,  the  examinee  fills-in  a  single  letter  abbreviation  for  each 
of  the  symbols  indicated  in  the  look-up  table. 


a  A  O  O  O 

r  t  c  d  p 


OcaaOO  aAdOADAOC^A  A  OAO 
- 


PS-Pattem  Recognition 


1.  Canceling  Symbols.  In  this  test,  the  examinee  searches  through  a  page  of  symbols,  and  places 
a  slash  through  the  target  symbols. 


Put  a  alaah  (  j  )  through  each  ^ 


'Oa 

_ _ _ _ _ iOOOOOD 

uDa::i>DX)C03a:>0300D<x)oo 


✓VNfS  - 1. 


2.  Finding  F  and¥.  In  this  test,  the  examinee  searches  for  the  conjunction  of  two  symbols,  and 
circles  each  “word”  that  contains  both  symbols. 


This  is  an  imaginary  language. 

Circle  every  word  that  contains  both  €  and  ¥  . 

T£/i9¥'*^<b  ^}=¥  "cfc*  «*e¥  e7=e  0y¥  §^»«£  J> 

^  e/i9¥c  §¥¥»  £¥€^J0MS¥£  ¥»0 

eniJ  Ac  ^dnQc  =»£  ♦P 

¥fifB/¥Q¥oe  £^■d¥  y«C  €«cf/i¥oe  d*«£M«f€  n*np  &JV 

■/3¥  *#0  £*p/9¥da0¥£  «¥♦  t  ♦{¥«»¥££  0y¥  ¥4C4»c; 


3.  Finding  a  and  t.  In  this  test,  the  examinee  circles  words  that  contain  both  an  ‘a’  and  a ‘t’. 


This  is  a  foreign  language. 

Circle  every  word  that  contains  both  a  and  t. 


car i she  accademiche(^overann^^liinita2loi^  armonie  nuovo  clima 
^ant^  nuovo  regola  ^^ta">naz  fonale  oitre  undici  anni  premi 
^hda^ ioni  ('amministrai^oni^atr imonial Q  saranno  voglia 
fascisticamente  regolati  sono  fierissimo  essere  stato  autore 
sollevanento  dello  prina  ciliegia  sollevamento  rivoluzionario 


PS-Scanning 

1.  Name  Comparison.  This  test  requires  that  the  examinee  check  each  pair  of  word  entries,  and 
indicate  whether  they  are  identical  or  whether  they  are  different  in  any  way. 


John  C.  Linder  )C  John  C.  Lender 
Investors  Syndicate  —  Investors  Syndicate 
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2.  Number  Comparison.  This  test  requires  that  the  examinee  check  each  pair  of  number  entries 
and  indicate  whether  they  are  identical  or  whether  they  differ  in  any  way. 


98765454  ~  98765454 

456712  X  456713 


3.  Clerical  Abilities  -  2.  In  this  test,  the  examinee  looks-up  corresponding  information  in  a  table 
and  then  uses  a  code  to  indicate  the  value  in  a  list  of  names. 


CREDIT  RATING 


AVERAGE  BALANCE 


Ada4iH.D.B. 

$2375.81 

Adamno,  B.  D. 

1M2.80 

All*.  J.F. 

287.45 

Andenon.  H.C. 

1712J0 

Andrews,  G.L. 

213020 

Annstrang.  C.  F. 

1543jrr 

Atwell,  CB. 

734.10 

B«i»i,M.V. 

5004.18 

BKtM.A.N. 

384.02 

BloaB)fieU.H.T. 

037.42 

Beswurtfay.  A.  P. 

1822.77 

Boni.X.L. 

187.10 

Brooton.  D.  H. 

478823 

Blown,  J.  R. 

00720 

Browne,  J.  B. 

1542B7 

Cerey,  R.E. 

$0144.14 

curk.av. 

318.80 

auk,c.v. 

1731.43 

Coeiwr.B.L. 

1135.58 

Corfaln.aF. 

30523 

Carwta.C.F. 

107477 

Coifiove.  J.  J. 

2507.18 

OMghlin.M.H. 

137404 

Cowsn,  F.L. 

341821 

CSine,  N.  D. 

05410 

Dele,  A.  M. 

801487 

Dely.l^E. 

235430 

Duiieti.R.T. 

1317.85 

Dewbctn,  F.  F. 

80522 

DMkcr.M.H. 

52720 

I  ges.99  or  1ms’  R 

IflOOM  to  S 

2,000.00  ormoie  T 

Nuns  Misting  0 


L  BAniM.M.V. 

2.  Botwortby,  A.  C. 

S.  CUA.CV. 

4  Botts.K.L. 

8.  Andrews,  L.  C. 
4  Dely.LE. 


7.  Cosgrove,  J.J. 

8.  Allen.  J.F. 
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TRACON  Training  Simulations.  Each  training  simulation  is  comprised  of  16 
overflights  and  departures  (with  roughly  equal  frequency),  and  12  arrivals.  The  planes  request 
entry  to  the  airspace  at  irregular  intervals  that  are  constrained  to  require  the  trainee  to  be 
always  oceupied  with  at  least  one  active  target.  The  trials  are  also  constrained  so  that  perfect 
performance  (handling  all  28  planes  sueeessfully)  is  just  at  the  skill  level  achieved  by  subject 
matter  experts.  Each  trial  is  concluded  in  30  min.  That  is,  in  order  to  provide  equivalent 
practice  time  across  subjects,  the  trials  are  ended  with  time  constraints,  rather  than  waiting 
until  all  planes  are  handled  -  which  otherwise  introduees  a  substantial  varianee  in  practice 
time. 

TRACON  Test  Simulations.  The  test  simulations  were  designed  to  be  broadly 
similar  to  the  training  simulations,  except  for  the  specific  requirements  of  the  altered  demands 
of  the  testing  phase.  That  is,  similar  to  the  training  simulations,  all  test  simulations  had  a 
total  of  28  planes  to  be  handled.  The  trials  were  designed  with  the  same  constraints  adopted 
in  the  training  simulations  (see  Appendix).  In  the  Arrival  test  simulations,  though,  no 
overflights  were  included.  Instead,  these  simulations  had  20  arrival  flights,  and  8  departures. 
In  the  Overflight  test  simulations,  there  were  no  arrival  flights,  but  20  overflights  and  8 
departures.  That  is,  the  simulations  were  designed  to  keep  a  common  demand  for  handling 
departures,  so  that  the  performanee  on  departure  flights  could  be  examined  for  spill-over 
effeets  from  the  other  manipulations,  aeross  the  Arrival  and  Overflight  conditions. 

The  other  test  simulation  manipulation,  which  was  envisioned  as  a  high  vs.  low 
working  memory/monitoring  manipulation,  was  a  contrast  between  Perfect  and  Error-Laden 
pilot  communications.  The  Perfect  condition  was  analogous  to  the  training  simulations,  where 
pilots  always  followed  the  operator’s  instructions  (at  least  to  the  capability  of  the  airplanes 
and  the  other  controllers).  For  example,  pilots  would  turn  as  instructed,  and  change  altitudes 
and  speed  as  instructed,  but  pilots  would  not  comply  with  an  instruction  to  reduce  speed 
below  “stall”  speed.  Also,  controllers  would  not  accept  “handoffs”  unless  the  airplanes  were 
within  radar  contact.  The  following  represent  visual  (in  the  Communieations  Box)  and 
auditory  (over  the  headset)  examples  of  Air  Traffic  Controller  (ATC)  —Pilot  communications 
in  the  Perfect  condition  (the  ATC  is  represented  as  “App/Dep”  for  Approach/Departure 
Controller;  the  pilot  responds  with  the  eall  letters/number  for  his/her  aircraft  and  a  read-back 
of  the  direetion): 

App/Dep:  N42A,  Change  altitude  to  2000. 

N42A:  Descending  to  2000. 

App/Dep:  C0445,  Turn  left  heading  035. 

C0445:  Roger.  Left  to  035. 

App/Dep:  WN077,  Cleared  direet  to  DRABB. 

WN077:  Going  direct  to  DRABB. 

App/Dep:  N782,  Cleared  for  ILS  approach.  Contact  tower  at  FAF. 

N782:  Thanks.  Good  day. 
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In  the  Error-Laden  pilot  communications,  approximately  30%  of  issued  commands 
resulted  in  a  communication  failure  or  a  mis-commimication.  Below  are  examples  of  these 
sequences: 

App/Dep:  N23F,  Turn  right  heading  295. 

CO903:  Roger.  Right  to  295. 

[Note  that  the  wrong  plane  acknowledged  and  followed  this  command] 

App/Dep:  N90S,  Change  altitude  to  2500. 

WN077:  Descending  to  2500. 

[Note  that  the  wrong  plane  acknowledged  and  followed  this  command] 

App/Dep:  N90S,  Turn  left  heading  355. 

N90S:  Repeat  that.  I  was  looking  at  a  map. 

App/Dep:  N23F,  Turn  left  heading  245. 

N23F:  Was  that  last  command  for  me? 

App/Dep:  N23G,  Turn  right  heading  325. 

[No  response] 

App/Dep:  N90S,  Turn  left  heading  360. 

N90S:  Say  again  please. 

App/Dep:  N39L,  Change  altitude  to  4500. 

N39L:  Your  signal  was  weak.  Say  again. 

Under  these  Error-Laden  conditions,  the  operator  must  carefully:  (a)  listen  to  the  read- 
back  of  the  command,  in  comparison  to  the  command  issued;  (b)  review  the  communication 
box  on  the  display  for  mis-matches  in  commands  and  readbacks;  and/or  (c)  monitor  the  radar 
display  for  airplanes  that  are  not  proceeding  as  instructed. 

By  parametrically  combining  Arrival  and  Overflight  simulations,  and  Perfect  and 
Error-Laden  conditions,  it  is  possible  to  derive  four  separate  testing  sets: 

1.  Arrival  -  Perfect 

2.  Arrival  -  Error-Laden 

3.  Overflight  -  Perfect 

4.  Overflight  -  Error-Laden 

It  is  important  to  keep  in  mind,  though,  that  each  of  these  test  conditions  also 
contained  8  departure  flights.  The  departure  flights  had  the  same  communication  conditions 
imposed,  to  avoid  having  the  participants  simply  choose  an  asymmetric  tradeoff  in  favor  of 
the  easier  departure  flights. 
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Procedure.  In  Session  1,  the  battery  of  21  ability  tests  was  administered  in  a 
four-hour  session,  with  5-minute  breaks  provided  after  eaeh  hour  of  testing.  Session  2  was 
devoted  to  a  70-minute  instructional  video  for  TRACON,  followed  by  three  30-minute 
TRACON  training  simulations.  Session  3  and  Session  4  contained  six  30-minute  TRACON 
training  simulations.  Session  5  contained  two  30-minute  TRACON  training  simulations, 
followed  by  four  30-minute  TRACON  test  simulations.  Sessions  6  and  7  eontained  six 
TRACON  test  simulations  eaeh.  Overall,  then,  there  was  8.5  hr.  of  TRACON  training  (17 
simulations),  and  8  hr.  of  TRACON  test  simulations  (16  simulations).  The  Test  simulations 
represented  a  2  x  2  x  4  within-subjects  design  (2  types  of  trials  -  Arrival  vs.  Overflight;  2 
types  of  communieations.  Perfect  and  Error-Laden,  and  4  repetitions  of  each  trial  type).  Both 
training  and  test  simulations  were  administered  in  counterbalaneed  orders. 

Results 

Training  Phase.  The  praetice  phase  of  TRACON  proeeeded  in  a  fashion  now 
generally  well  documented  in  our  previous  studies  (e.g.,  Ackerman,  1992;  Ackerman  & 
Kanfer,  1993;  Aekerman,  Kanfer,  &  Goff,  1995).  Mean  performanee  and  between-subjects 
standard  deviations  are  shown  for  overall  performance  (number  of  planes  handled  in  30- 
minute  simulations)  in  Figure  1.  In  addition,  mean  performance  levels  for  the  three 
eomponents  of  task  performance  —  Arrivals,  Overflights,  and  Departures  —  are  also  shown  in 
the  lower  panel  of  the  figure.  Performance  starts  off  relatively  poor,  with  an  average  of  8 
planes  handled  in  the  30-minute  simulation.  However,  with  practiee,  performance  quickly 
improves,  so  that  after  8.5  hr.  of  time-on-task,  partieipants  are,  on  average,  handling  nearly  20 
planes.  From  the  components-of-performance  perspeetive,  by  the  end  of  the  training  phase, 
the  average  participant  is  handling  60.3%  of  departures,  67.5%  of  arrivals,  and  80.5%  of 
overflights. 

In  order  to  smooth  the  data,  and  refine  the  total  volume  of  data  presented,  performance 
scores  were  collapsed  into  six  composite,  or  “session”  seores.  With  the  exception  of  Session 
6,  all  sessions  represent  an  average  of  3  simulation  trials.  (Session  6  only  contained  the  final 
two  training  simulation  trails).  Correlations  between  the  7  ability  eomposites  and  TRACON 
performance,  ineluding  overall  and  eomponent  measures,  for  each  of  the  6  sessions  are 
presented  in  Table  1.  In  addition,  the  eorrelations  among  the  ability  composites  are  provided 
at  the  bottom  of  the  table.  Although  there  is  a  substantial  amount  of  data  in  this  table,  there 
were  several  noteworthy  findings.  First,  the  prediction  of  overall  performanee  was  clearly 
inereased  from  the  first  training  session  to  the  final  training  session  for  several  abilities: 
Spatial,  Math,  PS-Complex,  and  Working  Memory.  In  contrast,  PS-Pattem  showed  negligible 
correlations  with  TRACON  performance  (consistent  with  the  results  obtained  by  Ackerman  & 
Cianeiolo,  under  review),  and  PS-Scanning  showed  signifieant  and  consistent  correlations  with 
performanee  aeross  praetiee.  Finally,  PS-Memory  showed  declining  correlations  with  practiee 
—  again  in  accordanee  with  previous,  smaller-scale  researeh  (Ackerman  &  Cianeiolo,  under 
review). 
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Trial  of  Practice  (30  min/trial) 

Figure!  TRACON  training  phase  performance.  Upper  panel:  overall  means  and  between- 
subjects  standard  deviations.  Lower  panel:  Means  by  task  component  (arrivals,  overflights, 
and  departures). 
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Table  1.  Correlations  between  Content,  Perceptual  Speed,  and 
TRACON  Training  Performance  (components  and  overall). 

Spatial  Math  PS  PS 

Complex  Memory 


Arrivals  1  .458** 

Arrivals  2  .588** 

Arrivals  3  .688** 

Arrivals  4  .698** 

Arrivals  5  .674** 

Arrivals  6  .719** 

Overflights  1  .377** 

Overflights  2  .445** 

Overflights  3  .532** 

Overflights  4  .534** 

Overflights  5  .556** 

Overflights  6  .596** 

Departures  1  .422** 

Departures  2  .439** 

Departures  3  .495** 

Departures  4  .441** 

Departures  5  .402** 

Departures  6  .497** 

Overall  1  .474** 

Overall  2  .585** 

Overall  3  .674** 

Overall  4  .657** 

Overall  5  .635** 

Overall  6  .700** 


395** 

.463** 

.236* 

500** 

.524** 

.205 

589** 

.588** 

.214* 

592** 

.595** 

.190 

567** 

.590** 

.186 

591** 

.591** 

.236* 

280** 

.332** 

.368** 

290** 

.380** 

.297** 

412** 

.308** 

376** 

.406** 

.241* 

496** 

.420** 

.292** 

493** 

.414** 

.144 

312** 

.338** 

.406** 

394** 

.416** 

.346** 

501** 

.533** 

.328** 

495** 

.445** 

.238* 

424** 

.423** 

.267* 

519** 

.507** 

.247* 

373** 

.430** 

379** 

476** 

.526** 

.313** 

587** 

.601** 

.304** 

574** 

.571** 

.244* 

571** 

.562** 

.266* 

614** 

.586** 

.242* 

Correlations  among  Ability  Predictors 


Spatial 

Math 

.607** 

PS-Complex 

.670** 

PS-Memory 

.379** 

PS-Pattem 

.187 

PS-Scanning 

.314** 

Working  Memory 

.489** 

*  p<  .05;  **  p<  .01 

.  N  = 

.590** 

.280** 

.554** 

.032 

.338** 

.492** 

.400** 

.632** 

.632** 

.545** 

.554** 

.414** 

Memory  Abilities  and 


PS 

PS 

Working 

Pattern 

Scanning 

Memory 

.032 

.240* 

.260* 

.018 

.271** 

.396** 

.063 

.341** 

.405** 

.082 

.304** 

.463** 

.143 

.322** 

.423** 

.146 

.304** 

.481** 

.111 

.231* 

.316** 

.041 

.233* 

.343** 

.087 

.295** 

.433** 

.060 

.248* 

.287** 

.089 

.270** 

.338** 

-.020 

.218* 

.337** 

.057 

.192 

.281** 

.031 

.215* 

.374** 

.010 

.272** 

.397** 

-.070 

.231* 

.331** 

.009 

.251* 

.306** 

-.019 

.232* 

.403** 

.076 

.251* 

.324** 

.033 

283** 

.436** 

.063 

.352** 

.466** 

.037 

.303** 

.430** 

.101 

.325** 

.416** 

.058 

.294** 

.472** 

.536** 

.124  .420** 
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These  patterns  of  correlations  with  overall  performance  were  largely  mirrored  in  the 
prediction  of  individual  task  components.  As  anticipated,  given  the  more  substantial 
processing  and  monitoring  demands  of  the  Arrival  component,  correlations  between  various 
ability  measures  and  performance  were  larger  for  Arrivals  than  for  either  Overflights  or 
Departures.  In  addition,  the  PS-Complex,  PS-Memory,  and  Working  Memory  correlations 
with  Overflights  were  smaller  than  the  respective  correlations  with  the  Departure  component, 
again  in  agreement  with  our  earlier  assertions  regarding  the  decreased  complexity  of 
Overflights  vs.  Departures. 

As  shown  in  the  lower  part  of  the  table,  there  is  substantial  common  variance  among 
the  ability  variables,  as  is  generally  expected.  However,  the  correlations  between  the  Working 
Memory  composite  and  the  other  abilities  is  somewhat  smaller  than  would  be  anticipated  if,  as 
claimed  by  Kyllonen  and  Christal  (1990),  working  memory  is  synonymous  with  reasoning  or 
general  intelligence.  In  fact,  with  a  general  ability  (g^)  composite  formed  from  Spatial,  Math, 
and  PS-Complex  composites.  Working  Memory  only  correlated  r  =  .608,  a  substantial  value, 
but  certainly  not  indicative  that  working  memory  and  g  are  substitutable  variables.  Given  the 
controversy  surrounding  the  degree  of  overlap  between  g  and  Working  Memory,  it  is  useful  to 
examine  the  respective  roles  of  these  two  composites  on  task  performance.  Figure  2  shows 
the  respective  ability-performance  correlations  over  TRACON  training  for  the  g  and  Working 
Memory  composites.  In  each  component  of  the  task,  the  g  measure  provides  a  higher  level  of 
predictive  validity  than  does  Working  Memory  for  TRACON  performance.  For  Overflights, 
though,  the  correlations  clearly  diverge,  with  virtually  stable  correlations  between  Working 
Memory  and  task  performance  over  training,  but  increasing  correlations  with  g.  (This 
divergence  is  seen  in  the  previous  table,  showing  the  increasing  correlations  with  between  the 
constituent  abilities  (Spatial,  Math,  and  PS-Complex)  and  Overflight  performance. 

Another  way  to  evaluate  the  overlap  between  g  and  Working  Memory  in  the  predictive 
context  is  to  conduct  a  series  of  multiple  regressions.  With  Overall  performance  at  initial  and 
final  training  sessions,  the  two  variables  (g  and  Working  Memory)  together  account  for  27.9% 
of  variance  in  TRACON  Session  1  performance.  If  g  is  first  entered  into  the  equation,  it 
accounts  for  27.4%  of  the  variance  (F(l,80)  =  30.09,  p  <  .01).  When  Working  Memory  is 
added  to  the  prediction  equation,  it  only  accounts  for  an  additional  0.5%  of  variance,  a  non¬ 
significant  effect  (F(l,79)  =  0.60).  In  contrast,  if  Working  memory  is  first  entered  into  the 
equation,  it  accounts  for  13.0%.  If  g  is  subsequently  added,  it  accounts  for  an  additional 
14.9%  of  the  variance,  a  highly  significant  addition  (F(l,79)  =  16.2,/?  <  .01). 

For  prediction  of  final  training  (TRACON  Session  6),  the  difference  between  g  and 
Working  Memory  as  predictors  is  even  more  striking.  Together,  they  account  for  54.3%  of 
the  TRACON  variance,  but  when  first  entered,  g  accounts  for  54.0%,  followed  by  Working 
Memory,  0.3  (F(l,79)  =  .52).  In  contrast,  if  Working  memory  is  first  entered  into  the 
equation,  it  accounts  for  21.9%.  If  g  is  subsequently  added,  it  accounts  for  an  additional 
32.4%  of  the  variance,  a  highly  significant  addition  (F(l,79)  =  56.1,  p  <  .01).  From  such 
analysis,  it  might  be  reasonable  to  conclude  that  Working  Memory  is  a  lot  less  than  g.  That 
is,  all  of  the  valid  variance  in  the  composite  of  Working  Memory  is  subsumed  in  the 
composite  of  Spatial,  Math,  and  PS-Complex  abilities. 


17 


h  0.7 


Figure  2.  Correlations  between  general/content  ability  [g]  and  Working  Memory  composites  and 
performance  on  TRACON  over  sessions  of  training  practice.  Lines  indicate  quadratic  curve-fits. 


18 


Another  perspective  that  can  (and  should)  be  taken  on  the  training  data,  is  to  answer 
the  question:  ‘How  well  can  we  predict  individual  differences  in  performance  at  final  level  of 
training?’  Using  the  single  g  composite,  54.0%  of  individual  differences  variance  in 
TRACON  Session  6  performance  was  accounted  for,  substantially  more  variance  than  is 
accounted  for  by  individual  differences  in  initial  performance  (Session  1)  on  TRACON 
(40.1%  shared  variance  between  Session  1  and  Session  6  performance  indicators).  Such 
results  are  consistent  with  Ackerman’s  (1988,  1989)  theory  of  the  ability  determinants  of 
skilled  performance.  The  increasing  correlation  between  g  and  task  performance  further 
refutes  the  Hulin,  Henry,  &  Noon  (1990,  see  also  Henry  &  Hulin,  1987)  claims  of  ubiquitous 
declines  in  predictive  validity  with  task  practice,  (r  =  .482  vs.  .729;  t  (90)  =  4.03,  p  <  .01). 
(See  Barrett,  Alexander,  and  Doverspike,  1992  for  a  more  extensive  critique  of  the  Hulin,  et 
al.  position). 

Summary.  The  preceding  discussion  supports  three  major  points,  as  follows: 

1.  TRACON  training  proceeded  as  expected,  in  terms  of  practice  effects  for  overall 
performance,  and  for  performance  components  (i.e.,  arrivals,  overflights,  departures). 

2.  Performance  on  TRACON  was  well-predicted  by  several  abilities,  but  a  composite  g 
variable  provided  an  excellent  accounting  for  performance,  especially  at  the  end  of 
training,  with  54.0%  of  interindividual  differences  variance  in  performance  accounted 
for. 

3.  Working  Memory,  although  significantly  and  substantially  correlated  with  TRACON 
training  phase  performance,  failed  to  account  for  any  variance  in  performance  over  and 
above  that  accounted  for  by  the  general  ability  composite  {g). 

Testing  Phase 

In  the  testing  phase  of  the  study,  trainees  were  subjected  to  16  simulations,  which 
made  up  a  complete  factorial  within-subjects  2x2x4  design.  The  factors  of  the  design  were 
Arrival/Overflight  (2  levels).  Perfect  vs.  Error-Laden  Communication  (2  levels)  and 
repetitions  (4  simulations  for  each  factor).  The  results  of  this  portion  of  the  study  are 
presented  in  two  sections:  overall  effects  (e.g.,  means)  and  ability  -  performance  correlations. 

Overall  Effects.  Mean  performance  and  standard  errors  of  the  means  for  the 
four  test  conditions  are  shown  in  Figure  3.  The  effect  of  Perfect  vs.  Error-Laden 
communication  was  clearly  substantial  for  performance  on  the  Arrival  conditions  (Mperfect  ^ 
14.39,  Merror-laden  “  11-44,  t(79)  =  14.51, />  <  .01).  The  effect  of  Perfect  vs.  Error-Laden 
communication  for  Overflight  conditions  was  significant,  though  somewhat  smaller  (Mperfect  “ 
17.75,  Merror-laden  ^  16.66,  t(79)  =  5.70,  p  <  .01),  as  anticipated.  An  analysis  of  the 
Departures  for  all  four  conditions  showed,  first  of  all,  that  fewer  departures  were  handled 
under  both  Arrival  conditions  (M  =  4.24)  than  under  the  Overflight  conditions  (M  =  5.87), 
(t(79)  =  -12.43,  p  <  .01).  There  was  clearly  a  spill-over  effect  of  the  Error-Laden 
communication  for  the  Departures  task  component  under  Arrivals  (Mperfect  ^  4.80,  Merror-laden 
=  3.65,  i(79)  =  9.24,  p  <  .01).  The  effect  of  Error-Laden  communication  on  Departures  under 
Overflight  conditions  was  essentially  negligible  (Mperfect  ^  Men-or-laden  ^  ^  ‘ 

.27,  ns). 
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Figure  3.  Mean  performance  on  TRACON  test  conditions.  Upper  panel:  Arrivals  vs.  Overflights. 
Lower  Panel:  Departures.  Bars  indicate  1  standard  error  of  the  mean. 
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In  general  the  manipulations  of  Arrivals  vs.  Overflights  had  the  desired  effects  on 
performance,  as  did  the  Perfect  vs.  Error-Laden  communication.  That  is,  as  reflected  in  both 
the  arrival  and  overflight  planes  handled,  and  the  departure  flights  handled  imder  the  various 
conditions,  the  most  difficult  condition  was  (1)  Arrivals-Error  Laden,  followed  by  (2) 
Arrivals-Perfect,  then  (3)  Overflights-Error  Laden,  and  finally  (4)  Overflights-Perfect.  In  fact, 
more  planes  [Overflights  +  Departures]  were  handled  imder  the  Overflights-Perfect  condition 
than  were  handled  in  the  final  Training  session  =  19.01  vs.  Moverflights-Perfect 

Condition  ^  22.49,  (^(80)  =  -8.87,  p  <  .01),  reflecting  again  that  overflights  are  easier  to  handle 
than  arrivals  are. 

Ability  -  Performance  Correlations 

Correlations  between  the  7  ability  composites  and  performance  under  the  four 
testing  conditions  are  shown  in  Table  2,  separately  for  the  Arrival/Overflight  performance 
conditions  and  the  Departure  performance  components.  In  addition,  tests  for  significant 
differences  between  the  respective  pairs  of  correlations  were  computed.  Significantly 
correlations  are  shown  by  the  arrows  in  the  table  (i.e.,  pairs  with  arrows  are  significantly 
different  in  magnitude  from  one  another).  Under  Perfect  communication  conditions,  only 
Spatial  and  PS-Complex  ability  composites  showed  significant  differences  between  Arrival 
and  Overflight  conditions,  with  the  larger  correlations  found  for  the  Arrival  conditions  (as 
expected  from  the  greater  demands  of  handling  arrivals).  In  contrast,  significant  differences 
were  found  for  all  7  ability  composites  between  Arrival  and  Overflight  conditions  under 
Error-Laden  communications,  again  with  the  higher  correlations  for  the  Arrival  conditions. 
Interestingly,  although  tending  to  show  slightly  higher  correlations  under  the  increased 
workload,  neither  the  differences  in  correlations  between  the  two  Arrival  conditions  (Perfect 
vs.  Error-Laden  communication)  nor  the  differences  between  the  two  Overflight  conditions 
(Perfect  vs.  Error-Laden  communication)  were  significant. 

In  contrast,  only  two  of  the  14  correlation  comparisons  for  Departures  were  significant, 
a  result  not  substantially  different  than  one  would  expect  by  chance  (given  the  per  comparison 
a  =  .05).  Thus,  even  though  mean  performance  on  the  Departure  components  clearly 
reflected  the  increased  load  on  the  Arrival  component  (see  the  mean  performance  levels  in 
Figure  3),  the  abilities  that  determine  performance  on  Departures  did  not  increase  in  influence 
in  tandem  with  the  increased  demands  of  the  task  overall. 

How  should  one  interpret  these  findings?  At  this  point,  the  best  representation  is  that 
large-scale  changes  in  task  demands  (such  as  Arrivals  vs.  Overflights  under  Error-Laden 
communications)  result  in  both  mean  performance  differences  and  increasing  demands  on  a 
wide  array  of  abilities.  (Indeed,  even  though  PS-Pattem  had  minimal  correlations  with 
TRACON  performance  during  training,  at  testing,  the  increasing  load  of  error-laden 
communication  brought  a  significant  difference  between  the  ability  correlations  with  Arrivals 
and  Overflights).  Other  manipulations,  such  as  the  effects  of  Arrivals  vs.  Overflights  or 
Perfect  vs.  Error-Laden  communications  on  Departures,  produced  mean  performance  effects, 
but  virtually  no  significant  differences  in  ability  determinants  of  task  performance. 
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Table  2.  Correlations  Between  Ability  Composites  and  TRACON  Test  performance. 
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Notes:  A  =  Arrivals,  O  =  Overflights,  100  =  Perfect  Communication,  70  =  Error-Laden  Communication 
Arrows  indicate  pairs  of  significantly  different  correlations.  *p  <  .05;  **p  <  .01 


Such  results  suggest  that  ability  determinants  of  performance  are  substantially  more  robust 
than  mean  performance  levels  when  manipulations  of  task  difficulty  are  implemented.  This  is 
a  critically  important  finding  -  one  that  supports  the  speculations  by  Ackerman  (1999)  that  it 
is  the  content  of  the  task,  rather  than  the  degree  of  information-processing  consistency  or  the 
degree  of  difficulty  of  the  task  that  most  affect  the  ability  -  performance  correlations. 

Summary  and  Conclusions 

In  this  research  project,  critical  task  constraints  (basic  processes)  and  critical  sources  of 
individual  differences  (abilities)  were  identified  and  evaluated  in  the  context  of  a  complex 
skill-learning  task.  Initial  training  on  the  air  traffic  control  simulation  task  (TRACON) 
established  a  baseline  set  of  ability  -  performance  correlation  patterns.  These  patterns 
indicated  that  the  task  is  well  predicted  by  a  tailored  selection  of  general/content  ability 
measures,  perceptual  speed  ability  measures,  and  working  memory  ability  measures.  By  the 
end  of  practice,  54%  of  the  between-subjects  variance  in  performance  was  accounted  for  by  a 
set  of  predictor  measures  administered  prior  to  task  practice.  This  predictability  of 
performance  from  ability  measures  exceeded  the  variance  accounted  for  by  measures  of  initial 
task  performance.  Performance  components  (arrivals,  departures,  and  overflights)  showed 
different  patterns  of  ability  -  performance  correlations,  in  accordance  with  prior  task  analysis 
specifications  of  the  basic  processes  that  underlie  each  task  component. 

Test  conditions  were  implemented  that  involved  modification  of  the  simulations  to 
either  place  greater  demands  on  spatial  abilities  (Arrivals  vs.  Overflights)  or  greater  demands 
on  working  memory,  task  monitoring,  and  decision  making  (Perfect  vs.  Error-Laden 
communications).  The  test  conditions  demonstrated  substantial  changes  to  overall 
performance,  in  accord  with  the  expectations  of  increased  task  processing  demands.  On  the 
one  hand,  the  manipulations  had  significant  direct  effects  on  several  ability  -  performance 
correlations,  also  in  accordance  with  theory  and  a  priori  predictions.  On  the  other  hand,  the 
respective  magnitudes  of  ability  -  performance  correlations  were  quite  robust  -  indicating  that 
even  substantial  changes  it  the  task  processing  demands  fail  to  fundamentally  change  the 
ability  determinants  of  individual  differences  in  task  performance. 

The  data  from  the  current  investigation,  in  concert  with  other  recent  findings  in  the 
literature,  suggest  two  major  conclusions.  First,  properly  selected  ability  measures  are 
powerful  predictors  of  individual  differences  in  complex  skill  performance.  The  prediction  of 
task  performance  can  remain  strong,  even  after  substantial  training  and  skill  acquisition. 
Second,  even  though  experimental  psychology  has  introduced  many  variables  that  have  an 
impact  on  mean  performance  levels  (e.g.,  memory  load),  ability  measures  are  robust  predictors 
of  individual  differences  in  task  performance  —  even  as  the  means  and  overall  variability 
change.  Broad  content  ability  measures,  for  example,  are  more  predictive  of  task  performance 
than  measures  of  elementary  information  processes,  which  presumably  map  directly  onto  task 
characteristics  (e.g.,  see  Ackerman,  199a;  Kyllonen,  1985). 
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From  an  applied  perspective,  these  findings  add  significantly  to  the  corpus  of  literature 
regarding  the  generalization  of  validity  of  ability  measures  for  predicting  task  and  job 
performance.  Even  though  these  results  do  not  necessarily  support  the  notion  that  a  single 
measure  of  general  intelligence  or  working  memory  can  be  an  optimal  predictor  of  task 
performance,  the  results  do  imply  that  properly  chosen  ability  measures  can  be  used  for 
applicant  selection  purposes  within  broader,  rather  than  narrow,  job  classes.  One  domain  of 
ability  assessment  that  has  been  insufficiently  investigated  in  the  past  five  decades  is  that  of 
perceptual  speed  abilities.  The  perceptual  speed  measures  used  in  the  current  study  and  in 
other  recent  studies  (e.g.,  Ackerman  &  Cianciolo,  1999)  may  offer  an  important  supplement  to 
extant  measures  of  content  and  general  abilities  in  situations  similar  to  selection  for  entry- 
level  technical  jobs,  such  as  those  in  the  U.S.  Air  Force. 
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Appendix.  Terminal  Radar  Approach  Control  (TRACON)  Simulation  Task 

The  task  used  for  this  research  is  a  uniquely  modified  Professional  version  (VI. 52)  of 
TRACON  simulation  software,  developed  by  Wesson  International.  Versions  of  this  program 
have  been,  and  are,  in  use  in  several  locations  in  this  country  (including  the  FAA,  NASA, 
DoD,  and  several  colleges  and  university  airway  sciences  programs)  for  training  of  air  traffic 
controllers.  Modifications  for  the  current  instantiation  of  the  program  allowed  for  the 
collection  of  a  variety  of  data,  described  in  more  detail  below. 

Task  analysis  is  a  critical  component  of  empirical  investigation  with  any  task,  but 
especially  with  a  complex  task  such  as  TRACON.  In  fact,  one  of  the  salient  reasons  for 
adoption  of  the  TRACON  task  here  is  that  there  has  been  extensive  historical  work  on  task 
analysis  for  the  full  range  of  Air  Traffic  Controller  tasks,  including  the  subset  implemented  in 
TRACON.  In  particular,  a  five-volume  set  of  task  analysis  materials  has  been  prepared  by  the 
FAA  (e.g.,  Ammerman,  et  al.  1987).  Additional  task  analyses  have  been  conducted  by  the 
U.S.  Air  Force,  and  by  HumRRO  (Means  et  al.,  1988).  All  of  these  have  been  carefully 
studied  in  an  effort  to  derive  the  critical  components  of  the  simulation  task  adopted  for 
empirical  evaluation. 

The  task  requires  that  subjects  learn  a  set  of  rules  for  positive  air  traffic  control, 
including  (a)  reading  flight  strips,  (b)  declarative  knowledge  about  radar  beacons,  airport 
locations,  airport  tower  handoff  procedures,  en-route  center  handoff  procedures,  (c)  plane 
separation  rules  and  procedures,  (d)  monitoring  strategies,  and  (e)  strategies  for  sequencing 
planes  for  maximum  efficient  and  safe  sector  traversal.  In  addition,  subjects  are  required  to 
acquire  human-computer  interface  skills:  including  issuing  mouse-based  commands,  menu 
retrieval,  keyboard  operations,  and  integration  between  visual  and  auditory  information 
channels.  Although  the  task  represents  a  substantial  reduction  of  rules  and  operational 
demands  in  comparison  to  the  real-world  job  of  an  Air  Traffic  Controller  (ATC),  it  represents 
an  excellent  simulation  vehicle  for  study  of  skill  acquisition,  within  a  time-frame  that  can  be 
handled  in  a  laboratory-based  research  environment. 

Display.  TRACON  presents  the  controller  (subject)  with  a  simulated  color  radar 
screen,  depicting  a  region  of  airspace,  radio  navigational  tower  locations  (VOR),  airports, 
sector  boundaries,  and  range  rings.  Planes  are  identified  by  an  icon  on  the  radar  scope,  with  a 
data  tag  that  indicates  plane  identification  and  altitude  information.  In  addition,  two  sets  of 
"Flight  Strips"  are  presented  at  the  right  side  of  the  display,  a  "Pending"  and  ^  "Active"  set. 
Each  flight  strip  contains  information  about  a  particular  flight,  including  identification 
information,  plane  type,  requested  speed  and  altitude,  and  sector  entry  and  exit  destination 
information  (See  Figure  Al).  Finally,  at  the  bottom  of  the  screen  is  a  "Communications  Box" 
-  which  shows  commands  issued  to  planes  (and  responses  by  the  pilots),  along  with  the 
controller’s  "score"  for  the  current  simulation. 

When  planes  are  about  to  enter  the  subject’s  sector  (at  a  boundary  or  on  the  runway  of 
an  airport)  this  information  is  announced  over  the  headset  (e.g.,  "Northwest  123  ready  for 
takeoff,"  or  "Delta  123,  with  you,  level  at  9,000").  However,  no  flight  is  allowed  to  cross  the 
sector  boundary  or  take  off  from  an  airport  without  explicit  authorization. 
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Figure  A1  (Preceding  Page).  Static  copy  of  TRACON®  screen.  There  are  three  major 
components  to  the  display.  The  right  hand  side  of  the  screen  shows  Pending  (not  under 
control)  and  Active  (under  control)  flight  strips.  Each  flight  strip  lists  (a)  plane  identifier,  (b) 
plane  type,  (c)  requested  speed,  (d)  requested  altitude,  (e)  Radar  fix  of  sector  entry,  (f)  Radar 
fix  of  sector  exit  (including  Tower  or  Center).  The  lower  part  of  the  screen  shows  a 
communications  box  that  gives  a  printout  of  the  current  (and  last  few)  commands  issued  by 
the  subject,  and  the  responses  from  pilots  or  other  controllers.  The  main  part  of  the  screen 
shows  a  radar  representation  of  the  Chicago  sector.  Planes  are  represented  by  a  plane  icon, 
and  a  data  tag  (which  gives  the  identifier,  the  altitude,  and  an  indication  of  current  changes  in 
altitude).  The  sector  is  bounded  by  the  irregular  dotted  polygon  describing  a  perimeter. 

Radar  fixes  are  shown  as  small  (+)  figures  on  the  radar  screen.  Airports  are  shown  with 
approach  cones,  and  a  circle  indicating  the  facility  proper.  A  continuous  radar  sweep  is 
shown  (updating  at  12  o’clock,  every  5  sec).  Range  rings  are  also  displayed,  indicating  5 
mile  distances. 
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Task  Controls  and  Knowledge  of  Results.  Subjects  interact  with  the  TRACON 
simulation  in  several  ways.  A  mouse  is  used  for  the  majority  of  input  activities,  although  the 
keyboard  was  also  used  alone,  or  in  conjunction  with  the  mouse.  For  each  plane  command,  a 
menu  of  command  choices  is  displayed  on  the  screen.  For  turns  (Left  or  Right),  a  small 
wheel  was  shown  so  that  a  direction  was  selected  by  pointing  to  the  appropriate  place  on  the 
wheel,  and  depressing  the  mouse  key.  Altitude  and  Speed  commands  resulted  in  the  display 
of  a  small  linear  display  from  which  the  subject  selected  a  particular  value.  Direct  and  Hold 
commands  require  the  subject  to  move  the  mouse-cursor  to  a  specific  VOR  fix  or  airport,  and 
select  a  location.  Resume  and  Handoff  commands  have  no  additional  menus,  but  are  initiated 
directly. 

Additional  commands  for  information  (Flight  Path,  Plane  Type,  and  Plane  Current 
Heading  (in  degrees)  and  Airspeed  (in  knots)  may  also  be  obtained.  Information  pertaining  to 
the  sector  constraints  (Map  of  VOR/Airport  fixes;  and  Airport  Information,  including  final 
approach  heading  and  altitude  requirements)  may  be  called  up  with  keyboard  commands. 

Knowledge  of  results  is  provided  visually  (by  text  in  the  communications  box)  and 
auditorially  with  a  read-back  by  the  pilot  or  other  controller  (using  digitized  speech  broadcast 
over  the  subject’s  headset).  If  a  command  is  not  allowed  (e.g.,  asking  a  pilot  to  increase  or 
decrease  speed  beyond  the  limitations  imposed  by  the  type  of  plane),  the  visual  and  auditory 
response  indicated  a  failure  to  comply  with  the  command  (e.g.,  "Sorry,  but  that  is  below  my 
’stall’  speed!").  Handoff  commands  differ  from  the  other  commands,  in  that  a  handoff  to 
another  sector  is  only  accepted  when  the  plane  is  within  5  miles  of  the  sector  boundary.  All 
other  requests  for  handoff  are  refused  by  Center  Controllers. 

In  addition,  planes  follow  (as  nearly  as  possible)  the  commands  issued  by  the  subject. 
Turn,  altitude  change,  and  speed  change  commands  are  processed  by  the  computer,  and  are 
carried  out  in  accordance  with  the  limitations  imposed  by  each  aircraft  type  (e.g.,  smaller 
planes  turned  in  a  smaller  radius  than  Boeing  747’s,  but  747’s  climbed  more  quickly  than  the 
smaller  planes).  Each  plane  performs  within  the  constraints  that  were  displayed  when  a 
subject  calls  up  the  information  for  that  plane  type. 

Finally,  when  errors  occur  (e.g.,  separation  conflicts,  near  misses,  crashes,  missed 
approaches,  handoff  errors),  additional  information  is  presented  to  the  subject.  In  each  of 
these  cases,  an  alert  circle  around  the  plane(s)  in  question  is  presented  on  the  screen,  and  a 
series  of  tones  are  presented  over  the  headset.  If  two  planes  crash,  a  message  appeared  on  the 
screen  indicating  which  of  the  planes  crashed.  (In  normal  training,  the  simulation  is 
immediately  halted  under  such  conditions.  However,  because  it  is  not  desirable  to  minimize 
learning  opportunities  of  subjects  who  have  crashes,  the  simulation  continues  under  such 
circumstances.) 

Points.  Subjects  are  told  to  perform  the  task  so  that  they  maximize  successful 
disposition  of  all  flight  paths,  but  that  safety  is  a  critical  component  of  the  task.  Points  are 
given  for  successful  accomplishment  of  each  plane’s  flight  plan,  and  penalty  points  are 
deducted  for  both  commission  or  omission  errors.  Points  assigned  are  based  on  a  priori 
judgements  of  task  component  difficulty  (e.g.,  arrivals  were  more  difficult  to  accomplish  than 
overflights,  so  arrivals  received  three  times  as  many  points).  The  point  assignments  are  used 
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to  encourage  subjects  to  develop  an  appropriate  strategy  for  task  component  emphasis. 


Trial  Description  Trials  for  the  task  are  created  and  pretested  to  be  roughly  equivalent 
in  difficulty.  Each  trial  contains  planes  that  are  divided  into  three  basic  categories 
(Overflights,  Departures,  and  Arrivals).  Overflights  are  planes  that  enter  and  exit  the  subject’s 
airspace  at  cruising  altitudes.  Subjects  are  required  to  acknowledge  these  airplanes  as  they 
approach  a  boundary  VOR  fix,  monitor  progress  through  the  sector,  and  handoff  to  a  "Center" 
controller.  Departures  are  planes  that  originate  at  one  of  the  four  airports,  climb  to  a  cruising 
altitude  and  are  handed  off  to  a  "Center"  controller.  Subjects  are  required  to  release 
departures  from  airports,  evaluate  and  remediate  potential  conflicts  as  the  planes  climb  to  a 
cruising  altitude  and  turn  to  intercept  their  intended  flight  paths,  and  then  handoff  planes  to 
the  appropriate  Center  controller.  Arrivals  enter  the  subject’s  airspace  from  one  of  the 
boundary  VOR  fixes,  and  have  to  be  landed  at  a  designated  airport.  Subjects  are  required  to 
direct  arrivals  onto  an  appropriate  heading  and  altitude  to  provide  an  acceptable  handoff  to  the 
appropriate  Tower  controller,  then  these  planes  can  land.  Practice  flights,  which  originate  at 
an  airport,  but  have  to  be  correctly  vectored  to  be  landed  again  at  the  same  airport  are 
classified  as  "Arrivals,"  because  demands  of  these  flights  are  most  similar  to  other  arrivals. 

For  all  flights,  the  subject  is  required  to  maintain  legal  separation  (at  least  1000  ft  in  altitude, 
or  3  mi  horizontally). 

A  successful  "handle"  of  a  flight  is  the  appropriate  accomplishment  of  the  respective 
flight  plan.  That  is,  for  a  departure  or  an  overflight,  the  accomplishment  was  a  successful 
handoff  to  the  appropriate  Center  controller.  For  a  landing,  the  accomplishment  was  the 
successful  landing  of  the  airplane. 

Errors  in  performance  take  a  variety  of  different  forms,  as  follows:  (1)  Incorrect  speed 
or  altitude  for  center  handoff;  (2)  A  failure  to  handoff  the  plane;  (3)  For  arrival  flights,  errors 
include  "wrong  approach  altitude,"  or  "wrong  approach  heading"  (which  requires  the  subject 
to  reorient  the  plane  for  another  landing  attempt);  and  (4)  Separation  conflicts,  near  misses, 
and  crashes  (for  a  differing  degrees  of  airspace  proximity  violations). 

Performance  Measures.  After  extensive  review  of  the  raw  data  from  initial 
experiments  (which  include  every  command  issued  by  subjects  during  TRACON  trials,  and  a 
series  of  summary  data  for  each  airplane  in  a  simulation,  and  for  each  simulation  overall),  a 
general  criterion  of  merit  has  been  selected  that  reflected  overall  task  performance.  This 
measure,  called  "Overall  Performance"  is  computed  as  the  sum  of  all  flights  accepted  into  the 
sector  that  have  a  final  disposition  within  the  simulation  time  (minus  any  planes  that  are 
incorrectly  disposed  of  —  e.g.,  crashes,  not-handed-off,  vectored  off  the  radar  screen).  This 
measure  is  generally  concordant  with  results  from  the  examination  of  the  criterion  space  for 
FAA  ATC  simulation  research  (e.g.,  see  Buckley,  DeBaryshe,  Hitchner,  &  Kohn,  1983). 

Other  measures  are  also  computed,  to  reflect  declarative  knowledge  (information  requests)  and 
task  component  processing  (separate  scores  for  number  of  arrivals,  departures,  and  overflights 
accepted  into  the  sector). 
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